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(54) Imaging system and method with dynamic brightness control 



(57) An imaging system with brightness control in- 
cludes an image capture subsystem and an image con- 
trol block and is adapted for use in conjunction with an 
image processing application. The image capture sub- 
system receives an image and converts this image into 
digital image data. The digital signal is then stored in a 
video buffer for access by the image control block. The 



image control block provides brightness control of an im- 
age sensor in the image control subsystem to optimize 
the brightness of the desired area of interest in the im- 
age relative to the background. In one application, the 
imaging system is used in conjunction with a lane track- 
ing system image processing application to detect road- 
way lane markings from a moving vehicle. 
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Description 

Field of the Invention 

[0001] The present invention relates to systems and 
methods which provide brightness control processing 
for images received from an image sensor. More partic- 
ularly, the present invention relates to imaging systems 
for optical vehicle safety systems or driver assistance 
systems. 

Background of the Invention 

[0002] Electronic imaging systems have long been 
used to provide images in various applications. One of 
the primary problems that imaging systems must over- 
come relates to variable scene brightness within the im- 
ager field-of-view. Because the brightness level in a 
scene to be imaged can often change dramatically, the 
image sensor must adjust the exposure/gain levels to 
compensate for the changing brightness level. Slow but 
very significant variations will obviously occur between 
day and night operation but more rapid variations may 
also occur. For example, one application of image sens- 
ing systems is in the field of vehicle safety or navigation 
systems. More particularly, image sensing systems can 
be used to monitor a vehicle's progress in a desired lane 
of traffic, and to issue a warning to the driver of the ve- 
hicle if it is unintentionally exiting the designated lane. 
In such applications, extreme and rapid brightness 
changes in the scene may be experienced due to tunnel 
exits, irregular shadows on the road, etc., along with the 
slower variations in brightness due to time of day and 
weather conditions. 

[0003] Typically, one of the problems encountered 
due to brightness variation in the scene to be imaged is 
that the image sensor only has a certain dynamic range. 
Without control over image sensor exposure/gain, ex- 
treme brightness will thus easily saturate the image sen- 
sor while an extremely dark scene will fall below the min- 
imum detectable light level. Therefore, one goal of the 
system must be to continually modify the exposure/gain 
settings of the image sensor in accordance with the 
changing brightness level in the scene. 
[0004] Typically, the maximum dynamic range of an 
imaging system is obtained by varying the exposure/ 
gain of the image sensor in the system. Such image sen- 
sor based systems are increasingly single chip VLSI de- 
vices. Therefore, in one approach to exposure/gain con- 
trol, the imager chip manufacturer permanently encodes 
an exposure/gain control algorithm on the chip. In such 
an approach, the on chip algorithm first counts all pixels 
in the image which have an intensity greater than a set 
threshold level of brightness. Based on this count, the 
on chip algorithm decides whether to increase or de- 
crease the amount of exposure/gain for the sensor. The 
on chip algorithm may alter the exposure/gain value by 
either large or small increments, depending on the rel- 



ative difference between the bright pixel count and the 
threshold values. 

[0005] One example of such a prior art imaging chip 
with on chip brightness compensation is the W5430 

5 Monolithic Sensor which is manufactured by VLSI Vi- 
sion, Ltd. in the United Kingdom. This chip counts all 
pixels in the image which have an intensity greater than 
a certain Very White Pixel threshold. This prior art ap- 
proach then changes the exposure and gain settings 

10 based on a comparison of this count to fixed brightness 
thresholds. 

[0006] In this prior art approach, the exposure/gain is 
optimized for an entire field-of-view. However, some ap- 
plications can require optimization only in small region 

15 (s) of interest in a scene. If the overall scene brightness 
is much different from the brightness in the region(s) of 
interest, the above-described dynamic exposure/gain 
control method can actually degrade the brightness and 
contrast in the region(s) of interest in an image. 

20 [0007] Another aspect of this prior art approach is that 
it employs one fixed set of brightness thresholds. These 
thresholds are chosen to provide images acceptable for 
most applications in most lighting conditions, but may 
provide poor quality images in extreme lighting condi- 

25 tions, such as glare, shadows, or overall dark or bright 
extremes of light. Some applications require images 
with optimal brightness and contrast even in these ex- 
treme conditions, in which case, the above-described 
method is not adequate. 

30 [0008] Also, this prior art approach is designed to ad- 
just for changes in brightness somewhat gradually, in 
an environment where brightness may be rapidly chang- 
ing, such adjustments allow for many image frames 
which are somewhat overexposed or underexposed. 

35 For example, glare or shadows moving through a scene 
may suddenly change the brightness of a portion of the 
scene very significantly. If the image system is being 
used in a computer system application where such glare 
or shadow effect is a factor, the image frames with less 

40 than optimal brightness and contrast may need to be 
either disregarded or processed further using algo- 
rithms for correcting severely degraded image quality 
which can significantly reduce system performance. 
[0009] Therefore, a need presently exists for a meth- 

45 od which provides dynamic brightness control of an im- 
age sensor and addresses the problems noted above. 

SUMMARY OF THE INVENTION 

50 [0010] The present invention provides an imaging 
system and method for brightness control of an image 
sensor which optimizes the dynamic range of the sensor 
for one or more regions of interest within the imager 
field-of-view. 

55 [0011] The present invention further provides an im- 
aging system and method for brightness control of an 
image sensor which optimizes exposure/gain of the sen- 
sor for a wide range of lighting conditions including very 
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dark or very bright. 

[0012] The present invention further provides an im- 
aging system and method for brightness control of an 
image sensor which adapts quickly to changing light 
conditions. 

[0013] In a preferred embodiment, the present inven- 
tion provides a method of brightness control of an image 
sensor based on image data received from the image 
sensor. One or more object(s) of interest within the im- 
age are identified and the brightness level of one or 
more region(s) in the image is detected, said region(s) 
comprising the background surrounding said object(s) 
of interest. An adjusted brightness level of the image is 
determined such that the brightness of the object(s) of 
interest will be optimized in relation to the background 
region(s) surrounding said object(s) of interest. The im- 
age sensor settings are controlled to alter the brightness 
level in the image region(s) to the adjusted brightness 
level to enhance imaging of the object(s) of interest. 
[0014] In another aspect, the present invention in- 
cludes an imaging system having an image sensor for 
providing image data of a scene. An image control block 
includes means for identifying one or more object(s) of 
interest in said image and means for detecting the 
brightness level of one or more region(s) in the image, 
said region(s) comprising the background surrounding * 
said object(s) of interest. The image control block also 
includes means for determining an adjusted brightness 
level of the image, such that the brightness of said object 
(s) of interest is optimized in relation to the background 
region(s) surrounding said object(s) of interest. The im- 
age control block also includes control means coupled 
to the image sensor, for controlling the image sensor to 
alter the brightness level of the image to the adjusted 
brightness level to enhance imaging of the object(s) of 
interest in the scene. 

[0015] A more complete understanding of the method 
and apparatus for an image sensor brightness control 
will be afforded to those skilled in the art, as well as a 
realization of additional advantages and objects thereof, 
by a consideration of the following detailed description 
of the preferred embodiments. Reference will be made 
to the appended sheets of drawings which will first be 
described briefly. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0016] 

Fig. 1 is a block diagram of the image brightness 
control method according to one embodiment of the 
present invention; 

Fig. 2 is a block diagram of the overall architecture 
of an optical lane departure warning system which 
includes the image brightness control system and 
method according to one embodiment of the 
present invention; and 



Figs. 3 and 4 are block diagrams illustrating the im- 
age brightness control system and method accord- 
ing to one embodiment of the present invention. 

5 DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0017] The present invention provides an imaging 
system and related method for brightness control of an 

10 image sensor. A preferred application is in an optical 
lane departure warning system, but the invention is not 
so limited. In the detailed description that follows, like 
element numerals are used to describe like elements 
illustrated in one or more of the above-mentioned fig- 

15 ures. 

[0018] Referring first to Fig. 1 , a high-level block dia- 
gram of the imaging system is depicted according to one 
embodiment of the present invention. Image capture 
subsystem 2 detects light within a scene and provides 

20 image data. Image brightness control block 33 operates 
on said image data using parameters which describe the 
location(s) and characteristics of one or more regions 
and objects of interest within a scene. Image brightness 
control block 33 can control an imager in the image cap- 

25 ture subsystem 2 to optimize the brightness and con- 
trast of the desired object(s) of interest within the scene 
relative to the scene background for the next frame or 
frames of image data. Image brightness control block 
33 includes, in one embodiment, a software implement- 

30 ed control that adjusts various threshold values em- 
ployed by said imager for exposure/gain setting adjust- 
ment. This control is illustrated by the feedback loop in 
Fig. 1. 

[0019] To provide a specific environment for discus- 
35 sion of the present invention, one application which will 
be described is a system which detects when a vehicle 
drifts out of a traffic lane and provides a warning to the 
driver of the vehicle. A block diagram of the overall ar- 
chitecture of a lane departure warning system is depict- 
40 ed in Fig. 2 according to one embodiment of the present 
invention. 

[0020] Power for the system is input into power sub- 
system 1 . Image capture subsystem 2 receives and de- 
tects incoming scene information to create image sig- 

45 nals and stores the image data in a buffer for use by 
processing subsystem 3. Processing subsystem 3 con- 
tains the microprocessor, memory, and software for 
analysis of the image data to determine the vehicle po- 
sition relative to the traffic lane and for the image bright- 

50 ness control processing. Processing subsystem 3 re- 
ceives status information from the vehicle and provides 
information to the vehicle and the driver concerning the 
warning system status. Processing subsystem 3 also 
provides information to image capture subsystem 2 to 

55 control the capture of succeeding images. 

[0021] Referring to Fig. 3, image capture subsystem 
2 and processing subsystem 3 are shown in more detail 
in one preferred embodiment of the present invention. 
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Image capture subsystem 2 preferably includes a lens 
21, an imager 22, and a video buffer 23. Imager 22 re- 
ceives the input image focused by lens 21 . Imager 22 
comprises an imaging device such as a CMOS device 
or a charge-coupled device (CCD). Such devices are 
commercially available and in a preferred embodiment, 
imager 22 may be a single chip imager such as a 
W5430 Monolithic Sensor manufactured by VLSI Vi- 
sion Ltd. Imager 22 sends a signal representing the in- 
put image to be stored in video buffer 23 where it may 
be accessed by processing subsystem 3. 
[0022] Also as shown in Fig. 3, processing subsystem 
3 includes, in one embodiment, a digital signal proces- 
sor (DSP) 31 and a field programmable gate array (FP- 
GA) 32. DSP 31 preferably implements software includ- 
ing image brightness control block 33, an input/output 
(I/O) processing block 34, and an image processing ap- 
plication block 35. As noted above, a lane tracking ap- 
plication is merely one example of an application of the 
present invention. FPGA 32 preferably implements the 
feedback loop between DSP 31 and imager 22. 
[0023] Image brightness control block 33 is, in one 
embodiment of the present invention, implemented in 
software and operates to determine the brightness of 
the background of the region(s) of interest in the scene 
from the image data provided by image capture subsys- 
tem 2 and to dynamically control imager 22 to optimize 
said brightness. More particularly, image brightness 
control block 33 adjusts various threshold values em- 
ployed by imager 22 for exposure/gain setting adjust- 
ment to optimize the brightness of the desired object(s) 
of interest relative to the background region(s) in the im- 
age including compensating for specific light conditions, 
such as extreme light or darkness. 
[0024] For example, when the region(s) of interest is 
(are) dark compared to the overall image or when the 
lighting condition is extremely dark, the thresholds are 
increased, resulting in the overall image becoming more 
bright while the region(s) of interest has(have) improved 
brightness and contrast. Similarly, when the region(s) of 
interest is(are) bright compared to the overall image or 
when the lighting condition is extremely bright, the 
thresholds are decreased, resulting in the overall image 
becoming more dark while the region(s) of interest has 
(have) improved brightness and contrast. In addition, 
the threshold values are set so that the exposure/gain 
can be quickly adapted to changes in lighting conditions. 
This also results in greater consistency and predictabil- 
ity of image brightness which allow the brightness con- 
trol method to perform more effectively. 
[0025] The steps as implemented by image bright- 
ness control block 33 are shown in more detail in the 
flow chart of Fig. 4. In operation, the image brightness 
control block 33 investigates one or more region(s) in 
the image which contain one or more objects of interest, 
said region(s) comprising the background surrounding 
said object(s) of interest. This investigation may option- 
ally include information provided by the image process- 
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ing application block 35 (as shown in Fig. 3). In one ap- 
plication of the present invention, the object(s) of inter- 
est can be roadway lane marker(s). 
[0026] The image brightness control block 33 seeks 

5 to maintain the maximum background level in the region 
(s) of interest in the image at a threshold, Tb, within a 
range, +/- Tw. At block 41, a histogram of pixel intensi- 
ties in each region of interest is compiled. As depicted 
at block 42, the number and intensity of image pixels 

10 comprising the object(s) of interest, Po, is estimated us- 
ing knowledge about the objects(s) of interest. The Po 
object pixels in the histogram are then disregarded. At 
block 43, the maximum value within the remaining his- 
togram is considered to be the maximum background 

15 intensity within each region, Mr. At block 44, the overall 
background maximum, Mb, is determined by identifying 
the maximum of all of the Mr. At block 45, the algorithm 
determines whether Mb is either within Tb +/- Tw or Tb 
+/- (3*Tw), makes any necessary adjustments to the ex- 

20 posure/gain control thresholds at block 46, and returns 
the adjusted threshold values to image capture subsys- 
tem 2 via a feedback loop as shown. 
[0027] More specifically, if Mb is within Tb +/- Tw, then 
the exposure/gain is considered "good", and the thresh- 

25 old values are not changed. If Mb is outside this ideal 
range, but still within Tb +/- (3*Tw), the exposure/gain 
is considered "fair", and the threshold values are 
changed by an amount proportional to (Tb-Mb). Finally, 
if Mb is outside the "fair" range, the exposure/gain is 

30 considered "poor", and the threshold values are adjust- 
ed by a maximum amount. In either case, the threshold 
values are increased when Mb < Tb and decreased 
when Mb > Tb. 

[0028] Whenever the threshold values are changed, 
35 a certain number of image frames are required to allow 
the imager 22 to adjust the exposure/gain based on the 
new threshold values. Therefore, a programmable 
frame delay is incorporated into the image brightness 
control loop so that the method described above is per- 
40 formed on every nth image frame. Without this delay, 
possible overcorrection and rebound can occur yielding 
images which cycle from overexposed to underexposed 
and back again. 

[0029] Lane tracking application block 35 also re- 
45 ceives the image data from image capture subsystem 2 
and attempts to identify and track roadway lane mark- 
ings. 

[0030] As shown in Fig. 3, the brightness control block 
33 and lane tracking application block 35 transmit their 

50 respective data to the input/output (I/O) processing 
block 34. The I/O processing block 34 then transmits 
this information through a feedback loop, which includes 
in one embodiment interface block 36 in the FPGA 32 
back to imager 22 in the image capture subsystem 2. 

55 [0031 ] As noted above, although the lane tracking ap- 
plication block 35 implements a preferred example of 
the present invention, the present invention is not so lim- 
ited. Suitable modifications may be made in the dynamic 
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brightness control block 33 described above to accom- 
modate such other applications. 
[0032] Having thus described a preferred embodi- 
ment of an image sensing system and method having 
dynamic brightness control, it should be apparent to 
those skilled in the art that certain advantages of the 
within system have been achieved. It should also be ap- 
preciated that various modifications, adaptations, and 
alternative embodiments thereof may be made within 
the scope and spirit of the present invention. The inven- 
tion is further defined by the following claims. 

Claims 

1 . A method of brightness control for an image sensor, 
said method comprising the steps of: 

receiving image data of an image from said im- 
age sensor; 

identifying one or more objects of interest in 
said image; 

detecting the brightness level of at least a por- 
tion of the image, said portion comprising the 
background image region(s) surrounding said 
object(s) of interest; 

determining an adjusted brightness level of 
said portion of the image, wherein the bright- 
ness of said object(s) of interest is optimized in 
relation to the background region(s) in the im- 
age surrounding the object(s) of interest; and 
controlling said image sensor to alter said 
brightness level of said portion of the image to 
said adjusted brightness level. 

2. A method as set out in claim 1 , wherein said portion 
of the image corresponds to the background. 

3. A method as set out in claim 1 , wherein said image 
sensor comprises an imaging chip having expo- 
sure/gain settings and wherein said controlling step 
alters said settings. 

4. A method as set out in claim 1 , wherein said image 
sensor comprises an imaging chip having expo- 
sure/gain settings and having an on chip exposure/ 
gain control algorithm wherein said controlling step 
alters parameters used by said on chip algorithm to 
effect alteration of said exposure/gain settings. 

5. An imaging system with dynamic brightness con- 
trol, said system comprising: an image sensor for 
providing image data of a scene; 

means for identifying one or more objects of in- 
terest in said image; 

means for detecting a brightness level of at 
least a portion of the image, said portion com- 



prising the background image region(s) sur- 
rounding said object(s) of interest; means for 
determining an adjusted brightness level of 
said portion of the image, wherein the bright- 

5 ness of said object(s) of interest is optimized in 

relation to the background region(s) in the im- 
age surrounding the object(s) of interest; and 
control means, coupled to the image sensor, for 
controlling the image sensor to alter said bright- 

10 ness level of said portion of the image to said 

adjusted brightness level. 

6. The imaging system as set out in claim 5, wherein 
said image sensor comprises: 

15 An imaging chip for detecting said image and 

providing image signals. 

7. The imaging system as set out in claim 6, wherein 
the imaging chip includes adjustable exposure/gain 

20 settings and wherein the control means adjusts the 
exposure/gain settings of the imaging chip. 

8. The imaging system as set out in claim 6, wherein 
the imaging chip includes adjustable exposure/gain 

25 settings and wherein the imaging chip further in- 
cludes an on chip exposure/gain control algorithm 
and wherein the control means adjusts the param- 
eters use by said on chip algorithm to effect altera- 
tion of the exposure/gain settings of the imaging 

30 chip. 

9. The imaging system as set out in claim 6, wherein 
said image sensor further includes a video buffer 
for storing digital image data. 

35 

10. The imaging system as set out in claim 5, wherein 
said portion of the image comprises the background 
portion surrounding the object(s) of interest and 
wherein said adjusted brightness level is chosen to 

40 maintain the brightness level of said background at 
or below a desired threshold. 

11. The imaging system as set out in claim 7, wherein 
the control means dynamically controls the imaging 

45 chip settings on a frame by frame basis. 

12. The imaging system as set out in claim 8, wherein 
the control means dynamically controls the imaging 
chip settings on a frame by frame basis. 

50 

13. The imaging system as set out in claim 5, wherein 
the means for detecting only examines the image 
data corresponding to a portion of the image data 
having a brightness level below a threshold value. 

55 
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